agora inbox for pgsql-hackers@postgresql.orghelp / color / mirror / Atom feed
[PATCH v29 11/11] Add documentations about Incremental View Maintenance 249+ messages / 2 participants [nested] [flat]
* [PATCH v29 11/11] Add documentations about Incremental View Maintenance @ 2019-12-20 01:25 Yugo Nagata <nagata@sraoss.co.jp> 0 siblings, 0 replies; 249+ messages in thread From: Yugo Nagata @ 2019-12-20 01:25 UTC (permalink / raw) --- doc/src/sgml/catalogs.sgml | 9 + .../sgml/ref/create_materialized_view.sgml | 124 ++++- .../sgml/ref/refresh_materialized_view.sgml | 8 +- doc/src/sgml/rules.sgml | 437 ++++++++++++++++++ doc/src/sgml/system-views.sgml | 9 + 5 files changed, 583 insertions(+), 4 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index d17ff51e28..3de3303cac 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -2224,6 +2224,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>relisivm</structfield> <type>bool</type> + </para> + <para> + True if relation is incrementally maintainable materialized view + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>relrewrite</structfield> <type>oid</type> diff --git a/doc/src/sgml/ref/create_materialized_view.sgml b/doc/src/sgml/ref/create_materialized_view.sgml index 0d2fea2b97..8c574062db 100644 --- a/doc/src/sgml/ref/create_materialized_view.sgml +++ b/doc/src/sgml/ref/create_materialized_view.sgml @@ -21,7 +21,7 @@ PostgreSQL documentation <refsynopsisdiv> <synopsis> -CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable> +CREATE [ INCREMENTAL ] MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable> [ (<replaceable>column_name</replaceable> [, ...] ) ] [ USING <replaceable class="parameter">method</replaceable> ] [ WITH ( <replaceable class="parameter">storage_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ] @@ -60,6 +60,125 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable> <title>Parameters</title> <variablelist> + <varlistentry> + <term><literal>INCREMENTAL</literal></term> + <listitem> + <para> + If specified, some triggers are automatically created so that the rows + of the materialized view are immediately updated when base tables of the + materialized view are updated. In general, this allows faster update of + the materialized view at a price of slower update of the base tables + because the triggers will be invoked. We call this form of materialized + view as "Incrementally Maintainable Materialized View" (IMMV). + </para> + <para> + When <acronym>IMMV</acronym> is defined without using <command>WITH NO DATA</command>, + a unique index is created on the view automatically if possible. If the view + definition query has a GROUP BY clause, a unique index is created on the columns + of GROUP BY expressions. Also, if the view has DISTINCT clause, a unique index + is created on all columns in the target list. Otherwise, if the view contains all + primary key attritubes of its base tables in the target list, a unique index is + created on these attritubes. In other cases, no index is created. + </para> + <para> + There are restrictions of query definitions allowed to use this + option. The following are supported in query definitions for IMMV: + <itemizedlist> + + <listitem> + <para> + Inner joins (including self-joins). + </para> + </listitem> + + <listitem> + <para> + Some built-in aggregate functions (count, sum, avg, min, max) without a HAVING + clause. + </para> + </listitem> + </itemizedlist> + + Unsupported queries with this option include the following: + + <itemizedlist> + <listitem> + <para> + Outer joins. + </para> + </listitem> + + <listitem> + <para> + Sub-queries. + </para> + </listitem> + + <listitem> + <para> + Aggregate functions other than built-in count, sum, avg, min and max. + </para> + </listitem> + <listitem> + <para> + Aggregate functions with a HAVING clause. + </para> + </listitem> + <listitem> + <para> + DISTINCT ON, WINDOW, VALUES, LIMIT and OFFSET clause. + </para> + </listitem> + </itemizedlist> + + Other restrictions include: + <itemizedlist> + + <listitem> + <para> + IMMVs must be based on simple base tables. It's not supported to + create them on top of views or materialized views. + </para> + </listitem> + + <listitem> + <para> + It is not supported to include system columns in an IMMV. + <programlisting> +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm02 AS SELECT i,j FROM mv_base_a WHERE xmin = '610'; +ERROR: system column is not supported with IVM + </programlisting> + </para> + </listitem> + + <listitem> + <para> + Non-immutable functions are not supported. + <programlisting> +CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm12 AS SELECT i,j FROM mv_base_a WHERE i = random()::int; +ERROR: functions in IMMV must be marked IMMUTABLE + </programlisting> + </para> + </listitem> + + <listitem> + <para> + IMMVs do not support expressions that contains aggregates + </para> + </listitem> + + <listitem> + <para> + Logical replication does not support IMMVs. + </para> + </listitem> + + </itemizedlist> + + </para> + </listitem> + </varlistentry> + <varlistentry> <term><literal>IF NOT EXISTS</literal></term> <listitem> @@ -155,7 +274,8 @@ CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] <replaceable>table_name</replaceable> This clause specifies whether or not the materialized view should be populated at creation time. If not, the materialized view will be flagged as unscannable and cannot be queried until <command>REFRESH - MATERIALIZED VIEW</command> is used. + MATERIALIZED VIEW</command> is used. Also, if the view is IMMV, + triggers for maintaining the view are not created. </para> </listitem> </varlistentry> diff --git a/doc/src/sgml/ref/refresh_materialized_view.sgml b/doc/src/sgml/ref/refresh_materialized_view.sgml index 675d6090f3..c29cfc19b6 100644 --- a/doc/src/sgml/ref/refresh_materialized_view.sgml +++ b/doc/src/sgml/ref/refresh_materialized_view.sgml @@ -35,9 +35,13 @@ REFRESH MATERIALIZED VIEW [ CONCURRENTLY ] <replaceable class="parameter">name</ owner of the materialized view. The old contents are discarded. If <literal>WITH DATA</literal> is specified (or defaults) the backing query is executed to provide the new data, and the materialized view is left in a - scannable state. If <literal>WITH NO DATA</literal> is specified no new + scannable state. If the view is an incrementally maintainable materialized + view (IMMV) and was unpopulated, triggers for maintaining the view are + created. Also, a unique index is created for IMMV if it is possible and the + view doesn't have that yet. + If <literal>WITH NO DATA</literal> is specified no new data is generated and the materialized view is left in an unscannable - state. + state. If the view is IMMV, the triggers are dropped. </para> <para> <literal>CONCURRENTLY</literal> and <literal>WITH NO DATA</literal> may not diff --git a/doc/src/sgml/rules.sgml b/doc/src/sgml/rules.sgml index d229b94d39..22e4cad103 100644 --- a/doc/src/sgml/rules.sgml +++ b/doc/src/sgml/rules.sgml @@ -1096,6 +1096,443 @@ SELECT word FROM words ORDER BY word <-> 'caterpiler' LIMIT 10; </sect1> +<sect1 id="rules-ivm"> +<title>Incremental View Maintenance</title> + +<indexterm zone="rules-ivm"> + <primary>incremental view maintenance</primary> +</indexterm> + +<indexterm zone="rules-ivm"> + <primary>materialized view</primary> + <secondary>incremental view maintenance</secondary> +</indexterm> + +<indexterm zone="rules-ivm"> + <primary>view</primary> + <secondary>incremental view maintenance</secondary> +</indexterm> + +<sect2 id="rules-ivm-overview"> +<title>Overview</title> + +<para> + Incremental View Maintenance (<acronym>IVM</acronym>) is a way to make + materialized views up-to-date in which only incremental changes are computed + and applied on views rather than recomputing the contents from scratch as + <command>REFRESH MATERIALIZED VIEW</command> does. <acronym>IVM</acronym> + can update materialized views more efficiently than recomputation when only + small parts of the view are changed. +</para> + +<para> + There are two approaches with regard to timing of view maintenance: + immediate and deferred. In immediate maintenance, views are updated in the + same transaction that its base table is modified. In deferred maintenance, + views are updated after the transaction is committed, for example, when the + view is accessed, as a response to user command like <command>REFRESH + MATERIALIZED VIEW</command>, or periodically in background, and so on. + <productname>PostgreSQL</productname> currently implements only a kind of + immediate maintenance, in which materialized views are updated immediately + in AFTER triggers when a base table is modified. +</para> + +<para> + To create materialized views supporting <acronym>IVM</acronym>, use the + <command>CREATE INCREMENTAL MATERIALIZED VIEW</command>, for example: +<programlisting> +CREATE <emphasis>INCREMENTAL</emphasis> MATERIALIZED VIEW mymatview AS SELECT * FROM mytab; +</programlisting> + When a materialized view is created with the <literal>INCREMENTAL</literal> + keyword, some triggers are automatically created so that the view's contents are + immediately updated when its base tables are modified. We call this form + of materialized view an Incrementally Maintainable Materialized View + (<acronym>IMMV</acronym>). +<programlisting> +postgres=# CREATE INCREMENTAL MATERIALIZED VIEW m AS SELECT * FROM t0; +NOTICE: could not create an index on materialized view "m" automatically +HINT: Create an index on the materialized view for effcient incremental maintenance. +SELECT 3 +postgres=# SELECT * FROM m; + i +--- + 1 + 2 + 3 +(3 rows) + +postgres=# INSERT INTO t0 VALUES (4); +INSERT 0 1 +postgres=# SELECT * FROM m; -- automatically updated + i +--- + 1 + 2 + 3 + 4 +(4 rows) +</programlisting> +</para> + +<para> + Some <acronym>IMMV</acronym>s have hidden columns which are added + automatically when a materialized view is created. Their name starts + with <literal>__ivm_</literal> and they contain information required + for maintaining the <acronym>IMMV</acronym>. Such columns are not visible + when the <acronym>IMMV</acronym> is accessed by <literal>SELECT *</literal> + but are visible if the column name is explicitly specified in the target + list. We can also see the hidden columns in <literal>\d</literal> + meta-commands of <command>psql</command> commands. +</para> + +<para> + In general, <acronym>IMMV</acronym>s allow faster updates of materialized + views at the price of slower updates to their base tables. Updates of + <acronym>IMMV</acronym> is slower because triggers will be invoked and the + view is updated in triggers per modification statement. +</para> + +<para> + For example, suppose a normal materialized view defined as below: + +<programlisting> +test=# CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm AS + SELECT a.aid, b.bid, a.abalance, b.bbalance + FROM pgbench_accounts a JOIN pgbench_branches b USING(bid); +SELECT 10000000 + +</programlisting> + + Updating a tuple in a base table of this materialized view is rapid but the + <command>REFRESH MATERIALIZED VIEW</command> command on this view takes a long time: + +<programlisting> +test=# UPDATE pgbench_accounts SET abalance = 1000 WHERE aid = 1; +UPDATE 1 +Time: 0.990 ms + +test=# REFRESH MATERIALIZED VIEW mv_normal ; +REFRESH MATERIALIZED VIEW +Time: 33533.952 ms (00:33.534) +</programlisting> +</para> + +<para> + On the other hand, after creating <acronym>IMMV</acronym> with the same view + definition as below: + +<programlisting> +test=# CREATE INCREMENTAL MATERIALIZED VIEW mv_ivm AS + SELECT a.aid, b.bid, a.abalance, b.bbalance + FROM pgbench_accounts a JOIN pgbench_branches b USING(bid); +test=# UPDATE pgbench_accounts SET abalance = 1000 WHERE aid = 1; +NOTICE: created index "mv_ivm_index" on materialized view "mv_ivm" +</programlisting> + + updating a tuple in a base table takes more than the normal view, + but its content is updated automatically and this is faster than the + <command>REFRESH MATERIALIZED VIEW</command> command. + +<programlisting> +test=# UPDATE pgbench_accounts SET abalance = 1000 WHERE aid = 1; +UPDATE 1 +Time: 13.068 ms +</programlisting> + +</para> + +<para> + Appropriate indexes on <acronym>IMMV</acronym>s are necessary for + efficient <acronym>IVM</acronym> because it looks for tuples to be + updated in <acronym>IMMV</acronym>. If there are no indexes, it + will take a long time. +</para> + +<para> + Therefore, when <acronym>IMMV</acronym> is defined, a unique index is created on the view + automatically if possible. If the view definition query has a GROUP BY clause, a unique + index is created on the columns of GROUP BY expressions. Also, if the view has DISTINCT + clause, a unique index is created on all columns in the target list. Otherwise, if the + view contains all primary key attritubes of its base tables in the target list, a unique + index is created on these attritubes. In other cases, no index is created. +</para> + +<para> + In the previous example, a unique index "mv_ivm_index" is created on aid and bid + columns of materialized view "mv_ivm", and this enables the rapid update of the view. + Dropping this index make updating the view take a loger time. +<programlisting> +test=# DROP INDEX mv_ivm_index; +DROP INDEX +Time: 67.081 ms + +test=# UPDATE pgbench_accounts SET abalance = 1000 WHERE aid = 1; +UPDATE 1 +Time: 16386.245 ms (00:16.386) +</programlisting> + +</para> + +<para> + <acronym>IVM</acronym> is effective when we want to keep a materialized + view up-to-date and small fraction of a base table is modified + infrequently. Due to the overhead of immediate maintenance, <acronym>IVM</acronym> + is not effective when a base table is modified frequently. Also, when a + large part of a base table is modified or large data is inserted into a + base table, <acronym>IVM</acronym> is not effective and the cost of + maintenance can be larger than the <command>REFRESH MATERIALIZED VIEW</command> + command. In such situation, we can use <command>REFRESH MATERIALIZED VIEW</command> + and specify <literal>WITH NO DATA</literal> to disable immediate + maintenance before modifying a base table. After a base table modification, + execute the <command>REFRESH MATERIALIZED VIEW</command> (with <literal>WITH DATA</literal>) + command to refresh the view data and enable immediate maintenance. +</para> + +</sect2> + +<sect2 id="rules-ivm-support"> +<title>Supported View Definitions and Restrictions</title> + +<para> + Currently, we can create <acronym>IMMV</acronym>s using inner joins, and some + aggregates. However, several restrictions apply to the definition of IMMV. +</para> + +<sect3 id="rules-ivm-support-joins"> +<title>Joins</title> +<para> + Inner joins including self-join are supported. Outer joins are not supported. +</para> +</sect3> + +<sect3 id="rules-ivm-support-aggregates"> +<title>Aggregates</title> +<para> + Supported aggregate functions are <function>count</function>, <function>sum</function>, + <function>avg</function>, <function>min</function>, and <function>max</function>. + Currently, only built-in aggregate functions are supported and user defined + aggregates cannot be used. When a base table is modified, the new aggregated + values are incrementally calculated using the old aggregated values and values + of related hidden columns stored in <acronym>IMMV</acronym>. +</para> + +<para> + Note that for <function>min</function> or <function>max</function>, the new values + could be re-calculated from base tables with regard to the affected groups when a + tuple containing the current minimal or maximal values are deleted from a base table. + Therefore, it can takes a long time to update an <acronym>IMMV</acronym> containing + these functions. +</para> + +<para> + Also note that using <function>sum</function> or <function>avg</function> on + <type>real</type> (<type>float4</type>) type or <type>double precision</type> + (<type>float8</type>) type in <acronym>IMMV</acronym> is unsafe. This is + because aggregated values in <acronym>IMMV</acronym> can become different from + results calculated from base tables due to the limited precision of these types. + To avoid this problem, use the <type>numeric</type> type instead. +</para> + + <sect4 id="rules-ivm-restrictions-aggregates"> + <title>Restrictions on Aggregates</title> + <para> + There are the following restrictions: + <itemizedlist> + <listitem> + <para> + If we have a <literal>GROUP BY</literal> clause, expressions specified in + <literal>GROUP BY</literal> must appear in the target list. This is + how tuples to be updated in the <acronym>IMMV</acronym> are identified. + These attributes are used as scan keys for searching tuples in the + <acronym>IMMV</acronym>, so indexes on them are required for efficient + <acronym>IVM</acronym>. + </para> + </listitem> + + <listitem> + <para> + <literal>HAVING</literal> clause cannot be used. + </para> + </listitem> + </itemizedlist> + </para> + </sect4> +</sect3> + +<sect3 id="rules-ivm-general-restricitons"> +<title>Other General Restrictions</title> +<para> + There are other restrictions which generally apply to <acronym>IMMV</acronym>: + <itemizedlist> + <listitem> + <para> + Sub-queries cannot be used. + </para> + </listitem> + + <listitem> + <para> + CTEs cannot be used. + </para> + </listitem> + + <listitem> + <para> + Window functions cannot be used. + </para> + </listitem> + + <listitem> + <para> + <acronym>IMMV</acronym>s must be based on simple base tables. It's not + supported to create them on top of views, materialized views, foreign tables, inhe. + </para> + </listitem> + + <listitem> + <para> + LIMIT and OFFSET clauses cannot be used. + </para> + </listitem> + + <listitem> + <para> + <acronym>IMMV</acronym>s cannot contain system columns. + </para> + </listitem> + + <listitem> + <para> + <acronym>IMMV</acronym>s cannot contain non-immutable functions. + </para> + </listitem> + + <listitem> + <para> + UNION/INTERSECT/EXCEPT clauses cannnot be used. + </para> + </listitem> + + <listitem> + <para> + DISTINCT ON clauses cannot be used. + </para> + </listitem> + + <listitem> + <para> + TABLESAMPLE parameter cannot be used. + </para> + </listitem> + + <listitem> + <para> + inheritance parent tables cannnot be used. + </para> + </listitem> + + <listitem> + <para> + VALUES clause cannnot be used. + </para> + </listitem> + + <listitem> + <para> + <literal>GROUPING SETS</literal> and <literal>FILTER</literal> clauses cannot be used. + </para> + </listitem> + + <listitem> + <para> + FOR UPDATE/SHARE cannot be used. + </para> + </listitem> + + <listitem> + <para> + targetlist cannot contain columns whose name start with <literal>__ivm_</literal>. + </para> + </listitem> + + <listitem> + <para> + targetlist cannot contain expressions which contain an aggregate in it. + </para> + </listitem> + + <listitem> + <para> + Logical replication is not supported, that is, even when a base table + at a publisher node is modified, <acronym>IMMV</acronym>s at subscriber + nodes are not updated. + </para> + </listitem> + + </itemizedlist> +</para> +</sect3> + +</sect2> + +<sect2 id="rules-ivm-distinct"> +<title><literal>DISTINCT</literal></title> + +<para> + <productname>PostgreSQL</productname> supports <acronym>IMMV</acronym> with + <literal>DISTINCT</literal>. For example, suppose a <acronym>IMMV</acronym> + defined with <literal>DISTINCT</literal> on a base table containing duplicate + tuples. When tuples are deleted from the base table, a tuple in the view is + deleted if and only if the multiplicity of the tuple becomes zero. Moreover, + when tuples are inserted into the base table, a tuple is inserted into the + view only if the same tuple doesn't already exist in it. +</para> + +<para> + Physically, an <acronym>IMMV</acronym> defined with <literal>DISTINCT</literal> + contains tuples after eliminating duplicates, and the multiplicity of each tuple + is stored in a hidden column named <literal>__ivm_count__</literal>. +</para> +</sect2> + +<sect2 id="rules-ivm-concurrent-transactions"> +<title>Concurrent Transactions</title> +<para> + Suppose an <acronym>IMMV</acronym> is defined on two base tables and each + table was modified in different a concurrent transaction simultaneously. + In the transaction which was committed first, <acronym>IMMV</acronym> can + be updated considering only the change which happened in this transaction. + On the other hand, in order to update the view correctly in the transaction + which was committed later, we need to know the changes occurred in + both transactions. For this reason, <literal>ExclusiveLock</literal> + is held on an <acronym>IMMV</acronym> immediately after a base table is + modified in <literal>READ COMMITTED</literal> mode to make sure that + the <acronym>IMMV</acronym> is updated in the latter transaction after + the former transaction is committed. In <literal>REPEATABLE READ</literal> + or <literal>SERIALIZABLE</literal> mode, an error is raised immediately + if lock acquisition fails because any changes which occurred in + other transactions are not be visible in these modes and + <acronym>IMMV</acronym> cannot be updated correctly in such situations. + However, as an exception if the view has only one base table and + <command>INSERT</command> is performed on the table, + the lock held on thew view is <literal>RowExclusiveLock</literal>. +</para> +</sect2> + +<sect2 id="rules-ivm-rls"> +<title>Row Level Security</title> +<para> + If some base tables have row level security policy, rows that are not visible + to the materialized view's owner are excluded from the result. In addition, such + rows are excluded as well when views are incrementally maintained. However, if a + new policy is defined or policies are changed after the materialized view was created, + the new policy will not be applied to the view contents. To apply the new policy, + you need to refresh materialized views. +</para> +</sect2> + +</sect1> + <sect1 id="rules-update"> <title>Rules on <command>INSERT</command>, <command>UPDATE</command>, and <command>DELETE</command></title> diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 2b35c2f91b..5366f707eb 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -1787,6 +1787,15 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>isimmv</structfield> <type>bool</type> + </para> + <para> + True if materialized view is incrementally maintainable + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>definition</structfield> <type>text</type> -- 2.25.1 --Multipart=_Mon__28_Aug_2023_16_05_30_+0900_b1OvQD_3A3ZMTGvj-- ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
* [PATCH v6a 2/5] gha: Andres' revisions @ 2026-06-01 19:09 Andres Freund <andres@anarazel.de> 0 siblings, 0 replies; 249+ messages in thread From: Andres Freund @ 2026-06-01 19:09 UTC (permalink / raw) --- .github/workflows/postgresql-ci.yml | 834 +++++++++++++++------------- 1 file changed, 447 insertions(+), 387 deletions(-) diff --git a/.github/workflows/postgresql-ci.yml b/.github/workflows/postgresql-ci.yml index a7ef0bee94d..e2795ca0ffb 100644 --- a/.github/workflows/postgresql-ci.yml +++ b/.github/workflows/postgresql-ci.yml @@ -4,6 +4,7 @@ name: GitHub Actions CI on: push: + # FIXME: Should we also run on PRs? # Restrict GITHUB_TOKEN to the minimum the jobs need: reading repo # contents during checkout. @@ -13,6 +14,7 @@ permissions: concurrency: group: ${{ github.workflow }}-${{ github.ref }} # Never cancel in-progress runs on master to ensure all commits are tested. + # FIXME: Should also not cancel REL_XY_STABLE cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} env: @@ -20,9 +22,19 @@ env: # concurrent jobs and retrying older runs have a chance of working. CLONE_DEPTH: 500 + # At the moment all jobs use 4vcore runners, and none seems to benefit from + # increasing concurrency further. + BUILD_JOBS: 4 + + # It's possible that some jobs benefit from an increased test concurrency, + # but a default of 4 is a safe bet. Individual jobs can override. + TEST_JOBS: 4 + CCACHE_MAXSIZE: "250M" + CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # check target for the autoconf builds + # Check target for the autoconf builds. Can be set to e.g. check to only + # only test the main regression tests. CHECK: check-world PROVE_FLAGS=--timer CHECKFLAGS: -Otarget @@ -30,6 +42,11 @@ env: # errors/warnings in one place. MBUILD_TARGET: all testprep MTEST_ARGS: --print-errorlogs --no-rebuild -C build + + # Can be set to a non-empty value to run a limited set of tests + # (e.g. --suite regress to only run the main regression tests). + MTEST_TARGET: + PGCTLTIMEOUT: 120 # avoids spurious failures during parallel tests TEMP_CONFIG: ${{ github.workspace }}/src/tools/ci/pg_ci_base.conf PG_TEST_EXTRA: kerberos ldap ssl libpq_encryption load_balance oauth @@ -79,20 +96,16 @@ env: --with-uuid=ossp --with-zstd - # Debian Trixie container image used by all Linux jobs. Built by + # Debian Trixie containers used by all Linux jobs. Built by # 'https://github.com/anarazel/pg-vm-images/';. - LINUX_CI_IMAGE: us-docker.pkg.dev/pg-ci-images/ci/linux_debian_trixie_ci:latest + CONTAINER_REPO: ghcr.io/anarazel/pg-vm-images/gha_main + CONTAINER_LINUX_CI: linux_debian_trixie_ci:latest + CONTAINER_LINUX_CI_DOCS: linux_debian_trixie_ci_docs:latest # The full set of OS / job selectors recognized by the `ci-os-only:` # commit-message directive parsed in the `setup` job below. CI_OS_ONLY_JOBS: "linux macos windows mingw compilerwarnings sanitycheck" - _LOG_PATHS: &log_paths | - build*/testrun/**/*.log - build*/testrun/**/*.diffs - build*/testrun/**/regress_log_* - build*/meson-logs/*.txt - jobs: # Parse "ci-os-only: ..." from the commit message and expose flags @@ -111,14 +124,26 @@ jobs: # Re-export workflow-level env vars that other jobs need to reference # from contexts (e.g. `jobs.<id>.container.image`) where the `env` # context is not available. - linux_ci_image: ${{ env.LINUX_CI_IMAGE }} + container_linux_ci: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI }} + container_linux_ci_docs: ${{ env.CONTAINER_REPO }}/${{ env.CONTAINER_LINUX_CI_DOCS }} steps: + # Anchor reused by other jobs further down. GitHub Actions supports YAML + # anchors/aliases but not merge keys, so the alias copies the whole step + # verbatim. The anchor is resolved at YAML parse time, so the alias + # keeps working even if this job were to be skipped at runtime. + - &nix_sysinfo_step + name: sysinfo + run: | + id + uname -a + ulimit -a -H && ulimit -a -S + env + - id: os env: MSG: ${{ github.event.head_commit.message }} shell: bash run: | - set -e all_os=${CI_OS_ONLY_JOBS} if printf '%s\n' "$MSG" | grep -qE '^ci-os-only: '; then sel=$(printf '%s\n' "$MSG" | sed -n 's/^ci-os-only: //p' | head -n 1) @@ -145,199 +170,22 @@ jobs: sanity-check: name: SanityCheck needs: setup - if: needs.setup.outputs.sanitycheck == 'true' + if: | + !cancelled() && + needs.setup.outputs.sanitycheck == 'true' runs-on: ubuntu-latest timeout-minutes: 15 - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + container: &linux_ci_container + image: ${{ needs.setup.outputs.container_linux_ci }} + + # Options passed to all linux containers. Not all of the jobs need + # all of them, but it's easier to just define them centrally. + # # --privileged is needed so the prepare step can write to sysctls # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern. - options: --privileged - env: - BUILD_JOBS: 8 - TEST_JOBS: 8 - CCACHE_DIR: ${{ github.workspace }}/ccache_dir - # no options enabled, should be small - CCACHE_MAXSIZE: "150M" - steps: - # Anchor reused by other jobs further down. GitHub Actions supports - # YAML anchors/aliases but not merge keys, so the alias copies the - # whole step verbatim. The anchor is resolved at YAML parse time, so the - # alias keeps working even if this job is skipped at runtime. - - &checkout_step - uses: actions/checkout@v6 - with: - fetch-depth: ${{ env.CLONE_DEPTH }} - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-sanitycheck-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-sanitycheck-${{ github.ref_name }}- - ccache-sanitycheck- - - - name: Prepare workspace - run: | - whoami - useradd -m postgres - chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" - - - name: Configure - run: | - su postgres <<-'EOF' - set -e - meson setup \ - --buildtype=debug \ - --auto-features=disabled \ - -Ddefault_library=shared \ - -Dtap_tests=enabled \ - build - EOF - - - name: Build - run: | - su postgres <<EOF - set -e - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - EOF - - # Run a minimal set of tests. The main regression tests take too long - # for this purpose. For now this is a random quick pg_regress style - # test, and a tap test that exercises both a frontend binary and the - # backend. - - name: Test - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - meson test ${MTEST_ARGS} --suite setup - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} \ - cube/regress pg_ctl/001_start_stop - EOF - - - name: Core backtraces - if: failure() - run: | - mkdir -m 770 /tmp/cores - find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \; - src/tools/ci/cores_backtrace.sh linux /tmp/cores - - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: sanitycheck-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore - - - # Build & test postgres on Linux in three configurations. - # - # Autoconf: - # - Uses address sanitizer (sanitizer failures are typically printed in - # the server log) - # - Configures postgres with a small segment size - # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range - # - # Meson: - # - Test both 64- and 32-bit builds - # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures - # are typically printed in the server log) - # - Uses io_method=io_uring - # - Uses meson feature autodetection - # - 32-bit build tests with LANG=C to give ICU some buildfarm-uncovered - # coverage. Also, newer Python insists on changing LC_CTYPE away from C, - # prevent that with PYTHONCOERCECLOCALE. - # - # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes - # print_stacktraces=1,verbosity=2, duh - # detect_leaks=0: too many uninteresting leak errors in short-lived binaries - linux: - name: Linux - ${{ matrix.name }} - needs: [setup, sanity-check] - if: | - !cancelled() && - needs.setup.outputs.linux == 'true' && - needs.sanity-check.result != 'failure' - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: false - matrix: - include: - - name: Autoconf - slug: autoconf - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=address - pg_test_pg_combinebackup_mode: '--copy-file-range' - configure: | - ./configure \ - --enable-cassert --enable-injection-points --enable-debug \ - --enable-tap-tests --enable-nls \ - --with-segsize-blocks=6 \ - --with-libnuma \ - --with-liburing \ - ${LINUX_CONFIGURE_FEATURES} \ - CLANG="ccache clang" - build: | - make -s -j${BUILD_JOBS} world-bin - test: | - make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} - logs_paths: | - **/*.log - **/*.diffs - **/regress_log_* - - - name: Meson (64-bit) - slug: meson-64 - cc: ccache gcc - cxx: ccache g++ - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - -Dllvm=enabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - - - name: Meson (32-bit) - slug: meson-32 - cc: ccache gcc -m32 - cxx: ccache g++ -m32 - sanitizer_flags: -fsanitize=alignment,undefined - pg_test_initdb_extra_opts: '-c io_method=io_uring' - configure: | - meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ - -Duuid=e2fs \ - --buildtype=debug \ - --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ - -DPERL=perl5.40-i386-linux-gnu \ - -Dlibnuma=disabled \ - build - build: | - ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} - ninja -C build -t missingdeps - test: | - PYTHONCOERCECLOCALE=0 LANG=C \ - meson test ${MTEST_ARGS} -C build --num-processes ${TEST_JOBS} - logs_paths: *log_paths - container: - image: ${{ needs.setup.outputs.linux_ci_image }} + # set kernel.core_pattern and (for the meson entries) to flip + # kernel.io_uring_disabled (default 2 on recent GH runner kernels). + # # Share the host PID + IPC namespaces. 017_shm.pl rapidly creates, # kill9's, and restarts postgres; with the container's small PID # space a new postgres can recycle the dead postmaster's PID before @@ -347,46 +195,36 @@ jobs: # # --ulimit raises memlock and core dump size. Memlock is needed for # running the AIO tests. - # - # --privileged is needed so the prepare step can write to sysctls - # under /proc/sys (it's mounted read-only without it). We use it to - # set kernel.core_pattern and (for the meson entries) to flip - # kernel.io_uring_disabled (default 2 on recent GH runner kernels). - options: --pid=host --ipc=host --ulimit memlock=-1:-1 --privileged + options: &linux_container_options | + --privileged --pid=host --ipc=host --ulimit memlock=-1:-1 env: - BUILD_JOBS: 4 - TEST_JOBS: 8 - CCACHE_DIR: /tmp/ccache_dir - DEBUGINFOD_URLS: "https://debuginfod.debian.net"; - - UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 - ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0:detect_stack_use_after_return=0 - CFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - CXXFLAGS: -Og -ggdb -fno-sanitize-recover=all ${{ matrix.sanitizer_flags }} - LDFLAGS: ${{ matrix.sanitizer_flags }} - CC: ${{ matrix.cc }} - CXX: ${{ matrix.cxx }} - - PG_TEST_INITDB_EXTRA_OPTS: ${{ matrix.pg_test_initdb_extra_opts }} - PG_TEST_PG_COMBINEBACKUP_MODE: ${{ matrix.pg_test_pg_combinebackup_mode }} + # no options enabled, should be small + CCACHE_MAXSIZE: "150M" steps: - - *checkout_step + - *nix_sysinfo_step - - name: Restore ccache - uses: actions/cache@v5 + - &checkout_step + uses: actions/checkout@v6 + with: + fetch-depth: ${{ env.CLONE_DEPTH }} + + - &ccache_restore_step + name: Restore ccache + id: ccache_restore + uses: actions/cache/restore@v5 with: path: ${{ env.CCACHE_DIR }} - key: ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}-${{ github.run_id }} + key: &ccache_key | + ccache-${{ github.job }}-${{ github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - ccache-linux-${{ matrix.slug }}-${{ github.ref_name }}- - ccache-linux-${{ matrix.slug }}- + ccache-${{ github.job }}-${{ github.ref_name }}- + ccache-${{ github.job }}- - - name: Prepare workspace + - &linux_prepare_workspace + name: Prepare workspace run: | useradd -m postgres chown -R postgres:postgres . - mkdir -p "$CCACHE_DIR" - chown -R postgres:postgres "$CCACHE_DIR" mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' @@ -400,41 +238,283 @@ jobs: 127.0.0.3 pg-loadbalancetest EOF + # By using a shell that includes su, the run commands themselves get + # simpler. As there are quite a few commands that need to use su... - name: Configure + shell: &su_postgres_shell | + su postgres -c "bash --noprofile --norc -eo pipefail {0}" run: | - su postgres <<EOF - set -e - ${{ matrix.configure }} - EOF + meson setup \ + --buildtype=debug \ + --auto-features=disabled \ + -Ddefault_library=shared \ + -Dtap_tests=enabled \ + build - name: Build - run: | - su postgres <<EOF - set -e - ${{ matrix.build }} - EOF + shell: *su_postgres_shell + run: &ninja_build_command | + ninja -C build -j${{env.BUILD_JOBS}} ${{env.MBUILD_TARGET}} + ninja -C build -t missingdeps - - name: Test world - run: | - su postgres <<EOF - set -e - ulimit -c unlimited - ${{ matrix.test }} - EOF + # FIXME: As long as we use per-run ccache caches, we should probably add + # a step that checks if there is sufficient new content to warrant + # saving the new cache. + - &ccache_save_step + name: Save ccache + uses: actions/cache/save@v5 + with: + path: ${{ env.CCACHE_DIR }} + key: ${{ steps.ccache_restore.outputs.cache-primary-key }} - - name: Core backtraces - if: failure() + # Run a minimal set of tests. The main regression tests take too long + # for this purpose. For now this is a random quick pg_regress style + # test, and a tap test that exercises both a frontend binary and the + # backend. + # + # To allow the command below to be reused by later tasks, we allow + # adding "setup" commands to be specified via the ADDITIONAL_SETUP + # environment variable. + # + # Note that this command is used on all platforms, therefore one needs + # to be careful about using only ${{env.}} variable references, + # linebreaks etc. + - name: Test + shell: *su_postgres_shell + env: + MTEST_TARGET: cube/regress pg_ctl/001_start_stop + run: &meson_test_world_cmd | + ${{case(runner.os == 'Windows', '', 'ulimit -c unlimited')}} + + ${{env.ADDITIONAL_SETUP}} + + echo ::group::test_setup + meson test ${{env.MTEST_ARGS}} --suite setup --logbase setup + echo ::endgroup:: + + meson test ${{env.MTEST_ARGS}} --num-processes ${{env.TEST_JOBS}} ${{env.MTEST_TARGET}} + + - &linux_collect_cores + name: Core backtraces + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh linux /tmp/cores - - name: Upload logs - if: failure() + # Note that this is used for both meson and autoconf builds + - &upload_logs_step + name: Upload logs + if: failure() && !cancelled() uses: actions/upload-artifact@v7 with: - name: linux-${{ matrix.slug }}-logs-${{ github.run_id }} - path: ${{ matrix.logs_paths }} + name: logs-${{ github.job }}-${{ github.run_id }}-${{ github.run_attempt }} + path: | + **/*.log + **/*.diffs + **/regress_log_* + **/crashlog-*.txt if-no-files-found: ignore + # Linux, Autoconf + # + # SPECIAL: + # - Uses address sanitizer (sanitizer failures are typically printed in + # the server log) + # - Configures postgres with a small segment size + # - Uses PG_TEST_PG_COMBINEBACKUP_MODE=--copy-file-range + linux-autoconf: + name: Linux - Autoconf + needs: [setup, sanity-check] + if: &linux_job_if | + !cancelled() && + needs.setup.outputs.linux == 'true' && + needs.sanity-check.result != 'failure' + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + + env: &linux_env + # Add both debian and linux, as symbols from the host can be visible during profiling + DEBUGINFOD_URLS: "https://debuginfod.debian.net https://debuginfod.ubuntu.com"; + # Use -O2 to reduce the test times, use -fno-sanitize-recover=all to make sanitizer test + # failures visible. + CFLAGS: -O2 -ggdb -fno-sanitize-recover=all + CXXFLAGS: -O2 -ggdb -fno-sanitize-recover=all + LDFLAGS: + CC: ccache gcc + CXX: ccache g++ + CLANG: ccache clang + + # Configure sanitizer runtime behavior to be suitable for running tests: + # disable_coredump=0, abort_on_error=1: for useful backtraces in case of crashes + # print_stacktraces=1,verbosity=2, duh + # detect_leaks=0: too many uninteresting leak errors in short-lived binaries + UBSAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:verbosity=2 + ASAN_OPTIONS: print_stacktrace=1:disable_coredump=0:abort_on_error=1:detect_leaks=0 + + steps: + # GitHub Actions does not make it easy to share some, but not all, + # environment variables between related tasks. We solve that for the + # linux- tasks by updating the environment variables programmatically. + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=address + PG_TEST_PG_COMBINEBACKUP_MODE: --copy-file-range + run: &linux_update_config_cmd | + echo "CFLAGS=$CFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "CXXFLAGS=$CXXFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + echo "LDFLAGS=$LDFLAGS ${SANITIZER_FLAGS}" >> "$GITHUB_ENV" + + echo "CC=${CC}" >> "$GITHUB_ENV" + echo "CXX=${CXX}" >> "$GITHUB_ENV" + + echo "PG_TEST_INITDB_EXTRA_OPTS=${PG_TEST_INITDB_EXTRA_OPTS}" >> "$GITHUB_ENV" + echo "PG_TEST_PG_COMBINEBACKUP_MODE=${PG_TEST_PG_COMBINEBACKUP_MODE}" >> "$GITHUB_ENV" + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + ./configure \ + --enable-cassert --enable-injection-points --enable-debug \ + --enable-tap-tests --enable-nls \ + --with-segsize-blocks=6 \ + --with-libnuma \ + --with-liburing \ + ${LINUX_CONFIGURE_FEATURES} + + - name: Build + shell: *su_postgres_shell + run: | + make -s -j${BUILD_JOBS} world-bin + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: | + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + + - *linux_collect_cores + - *upload_logs_step + + + # Linux Meson, 32 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + # - tests with LANG=C to give ICU some buildfarm-uncovered coverage. Also, + # newer Python insists on changing LC_CTYPE away from C, prevent that with + # PYTHONCOERCECLOCALE. + linux-meson-32: + name: Linux - Meson (32-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + CC: ccache gcc -m32 + CXX: ccache g++ -m32 + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + --pkg-config-path /usr/lib/i386-linux-gnu/pkgconfig/ \ + -DPERL=perl5.40-i386-linux-gnu \ + -Dlibnuma=disabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + env: + PYTHONCOERCECLOCALE: 0 + LANG: C + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # Linux Meson, 64 bit + # + # SPECIAL: + # - Uses undefined behaviour and alignment sanitizers, (sanitizer failures + # are typically printed in the server log) + # - Uses io_method=io_uring + # - Uses meson feature autodetection + linux-meson-64: + name: Linux - Meson (64-bit) + needs: [setup, sanity-check] + if: *linux_job_if + runs-on: ubuntu-latest + container: *linux_ci_container + timeout-minutes: 60 + env: *linux_env + + steps: + - name: Update Environment + env: + SANITIZER_FLAGS: -fsanitize=alignment,undefined + PG_TEST_INITDB_EXTRA_OPTS: -c io_method=io_uring + run: *linux_update_config_cmd + + - *nix_sysinfo_step + - *checkout_step + - *ccache_restore_step + - *linux_prepare_workspace + + - name: Configure + shell: *su_postgres_shell + run: | + meson setup \ + ${MESON_COMMON_PG_CONFIG_ARGS} \ + -Duuid=e2fs \ + --buildtype=debug \ + -Dllvm=enabled \ + build + + - name: Build + shell: *su_postgres_shell + run: *ninja_build_command + + - *ccache_save_step + + - name: Test world + shell: *su_postgres_shell + run: *meson_test_world_cmd + + - *linux_collect_cores + - *upload_logs_step + + # SPECIAL: # - Enables --clone for pg_upgrade and pg_combinebackup # - Specifies configuration options that test reading/writing/copying of node trees @@ -449,13 +529,6 @@ jobs: runs-on: macos-15 timeout-minutes: 60 env: - BUILD_JOBS: 4 - # Test performance regresses noticeably when using all cores. 8 works OK. - # https://postgr.es/m/20220927040208.l3shfcidovpzqxfh%40awork3.anarazel.de - # Fix: Needs to be re-tested for GitHub Actions. - TEST_JOBS: 8 - - CCACHE_DIR: ${{ github.workspace }}/ccache_dir MACPORTS_CACHE: ${{ github.workspace }}/macports-cache MESON_FEATURES: >- @@ -497,44 +570,28 @@ jobs: -c debug_parallel_query=regress steps: - - *checkout_step + - *nix_sysinfo_step - - name: Sysinfo - run: | - id - uname -a - ulimit -a -H && ulimit -a -S - env + - *checkout_step - name: Setup core files run: | mkdir -p $HOME/cores sudo sysctl kern.corefile="$HOME/cores/core.%P" - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-macos-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-macos-${{ github.ref_name }}- - ccache-macos- - - - name: Compute MacPorts cache key + - name: "Macports: Compute cache key" id: mpkey run: | macos_major=$(sw_vers -productVersion | sed 's/\..*//') pkglist_hash=$(printf '%s' "$MACOS_PACKAGE_LIST" | md5 -q) script_hash=$(md5 -q src/tools/ci/ci_macports_packages.sh) - echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}-${GITHUB_RUN_ID}" >> "$GITHUB_OUTPUT" - echo "restore-key=macports-${macos_major}-${pkglist_hash}-${script_hash}-" >> "$GITHUB_OUTPUT" + echo "key=macports-${macos_major}-${pkglist_hash}-${script_hash}" >> "$GITHUB_OUTPUT" - - name: Restore MacPorts cache + - name: "MacPorts: Restore cache" uses: actions/cache@v5 with: path: ${{ env.MACPORTS_CACHE }} key: ${{ steps.mpkey.outputs.key }} - restore-keys: ${{ steps.mpkey.outputs.restore-key }} # Use MacPorts, even though Homebrew is installed. The installation # of the additional packages we need would take quite a while with @@ -546,7 +603,7 @@ jobs: # the large MacPort tree around to figure out that p5-io-tty is # actually p5.34-io-tty. Using the unversioned name works, but # updates MacPorts every time. - - name: Install dependencies (MacPorts) + - name: "MacPorts: Install dependencies" env: # Pass token so the script's GitHub API call to list MacPorts # releases isn't subject to the 60/h/IP unauthenticated rate @@ -560,11 +617,14 @@ jobs: echo /opt/local/sbin >> "$GITHUB_PATH" echo /opt/local/bin >> "$GITHUB_PATH" + - *ccache_restore_step + - name: Configure + env: + PKG_CONFIG_PATH: /opt/local/lib/pkgconfig/ run: | - export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig/" meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ --buildtype=debug \ -Dextra_include_dirs=/opt/local/include \ -Dextra_lib_dirs=/opt/local/lib \ @@ -574,25 +634,21 @@ jobs: build - name: Build - run: ninja -C build -j${BUILD_JOBS} ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: | - ulimit -c unlimited # default is 0 - ulimit -n 1024 # default is 256, pretty low - meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + env: + # default is 256, pretty low + ADDITIONAL_SETUP: ulimit -n 1024 + run: *meson_test_world_cmd - name: Core backtraces - if: failure() + if: failure() && !cancelled() run: src/tools/ci/cores_backtrace.sh macos "$HOME/cores" - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: macos-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-vs: @@ -605,10 +661,10 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 8 # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' + TAR: "c:/windows/system32/tar.exe" MESON_FEATURES: >- -Dcpp_args=/std:c++20 @@ -618,13 +674,13 @@ jobs: -Dssl=openssl -Dplperl=enabled -Dplpython=enabled - TAR: "c:/windows/system32/tar.exe" defaults: run: shell: cmd steps: - - name: Disable Windows Defender + - &windows_disable_defender + name: Disable Windows Defender shell: powershell run: | Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable @@ -723,33 +779,36 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - name: Configure run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson setup --backend ninja %MESON_COMMON_PG_CONFIG_ARGS% %MESON_FEATURES% --buildtype debug -Db_pch=true -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include -DTAR=%TAR% build + meson setup ^ + --backend ninja ^ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} ^ + ${{env.MESON_FEATURES}} ^ + --buildtype debug ^ + -Db_pch=true ^ + -Dextra_lib_dirs=d:\openssl\1.1\lib -Dextra_include_dirs=d:\openssl\1.1\include ^ + -DTAR=${{env.TAR}} ^ + build - name: Build run: | call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - ninja -C build %MBUILD_TARGET% + ninja -C build ${{env.MBUILD_TARGET}} ninja -C build -t missingdeps - name: Test world - run: | - call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 - meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + env: + ADDITIONAL_SETUP: | + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-vs-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step windows-mingw: @@ -762,7 +821,7 @@ jobs: runs-on: windows-2022 timeout-minutes: 60 env: - TEST_JOBS: 4 # higher concurrency causes occasional failures + # Avoid port conflicts between concurrent tap tests PG_TEST_USE_UNIX_SOCKETS: 1 PG_REGRESS_SOCK_DIR: 'd:\pgsock' TAR: "c:/windows/system32/tar.exe" @@ -776,7 +835,6 @@ jobs: MESON_FEATURES: >- -Dnls=disabled - CCACHE_DIR: D:/a/ccache CCACHE_MAXSIZE: "500M" CCACHE_SLOPPINESS: pch_defines,time_macros CCACHE_DEPEND: 1 @@ -786,17 +844,7 @@ jobs: shell: 'D:\msys64\usr\bin\bash.exe --login -eo pipefail "{0}"' steps: - - name: Disable Windows Defender - shell: powershell - run: | - Set-MpPreference -DisableRealtimeMonitoring $true -SubmitSamplesConsent NeverSend -MAPSReporting Disable - # Verify Defender status - $status = Get-MpComputerStatus -ErrorAction SilentlyContinue - if ($status) { - Write-Host "RealTimeProtectionEnabled: $($status.RealTimeProtectionEnabled)" - Write-Host "AntivirusEnabled: $($status.AntivirusEnabled)" - } - + - *windows_disable_defender - *checkout_step # Relocate the preinstalled MSYS2 tree from C:\ (slow system disk) to @@ -835,6 +883,8 @@ jobs: ${MINGW_PACKAGE_PREFIX}-readline \ ${MINGW_PACKAGE_PREFIX}-zlib + - *nix_sysinfo_step + - name: Install additional dependencies run: | # Pin IPC::Run to NJM/IPC-Run-20250809.0; TODDR/IPC-Run-20260322.0 @@ -845,42 +895,32 @@ jobs: - name: Setup socket directory shell: cmd - run: mkdir %PG_REGRESS_SOCK_DIR% + run: mkdir ${{env.PG_REGRESS_SOCK_DIR}} - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-mingw-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-mingw-${{ github.ref_name }}- - ccache-mingw- + - *ccache_restore_step - name: Configure run: | meson setup \ - ${MESON_COMMON_PG_CONFIG_ARGS} \ + ${{env.MESON_COMMON_PG_CONFIG_ARGS}} \ -Ddebug=true -Doptimization=g -Db_pch=true \ - ${MESON_COMMON_FEATURES} \ - ${MESON_FEATURES} \ - -DTAR=${TAR} \ + ${{env.MESON_COMMON_FEATURES}} \ + ${{env.MESON_FEATURES}} \ + -DTAR=${{env.TAR}} \ build - name: Build - run: ninja -C build ${MBUILD_TARGET} + run: *ninja_build_command + + - *ccache_save_step - name: Test world - run: meson test ${MTEST_ARGS} --num-processes ${TEST_JOBS} + run: *meson_test_world_cmd # FIX: We need to collect crashlogs but they are not collected. cdb.exe # is installed on the runner so it needs to be configured. - - name: Upload logs - if: failure() - uses: actions/upload-artifact@v7 - with: - name: windows-mingw-logs-${{ github.run_id }} - path: *log_paths - if-no-files-found: ignore + - *upload_logs_step + # Test that code can be built with both gcc and clang without warnings, # with various combinations of cassert/dtrace flags. Trace probes have @@ -900,24 +940,12 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 60 container: - image: ${{ needs.setup.outputs.linux_ci_image }} + image: ${{ needs.setup.outputs.container_linux_ci_docs }} env: - BUILD_JOBS: 4 - CCACHE_DIR: /tmp/ccache_dir # Use larger ccache cache as this job compiles with multiple # compilers / flag combinations. CCACHE_MAXSIZE: "1G" steps: - - *checkout_step - - - name: Restore ccache - uses: actions/cache@v5 - with: - path: ${{ env.CCACHE_DIR }} - key: ccache-compiler-warnings-${{ github.ref_name }}-${{ github.run_id }} - restore-keys: | - ccache-compiler-warnings-${{ github.ref_name }}- - ccache-compiler-warnings- - name: Sysinfo run: | @@ -929,83 +957,108 @@ jobs: clang -v env + - *checkout_step + + - *ccache_restore_step + - name: Setup workspace run: | echo "COPT=-Werror" > src/Makefile.custom - mkdir -p "$CCACHE_DIR" # gcc, cassert off, dtrace on - name: gcc warnings + (dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # gcc, cassert on, dtrace off - name: gcc warnings + (cassert) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ --enable-cassert \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin # clang, cassert off, dtrace off - name: clang warnings - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + # clang, cassert on, dtrace on - name: clang warnings + (cassert + dtrace) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache clang.cache \ --enable-cassert \ --enable-dtrace \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + - name: mingw warnings (cross compilation) - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --host=x86_64-w64-mingw32ucrt \ --enable-cassert \ --without-icu \ CC="ccache x86_64-w64-mingw32ucrt-gcc" \ CXX="ccache x86_64-w64-mingw32ucrt-g++" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} world-bin + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} world-bin + ### # Verify docs can be built ### # XXX: Only do this if there have been changes in doc/ since last build - name: Build documentation - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ --cache gcc.cache \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -C doc + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -C doc ### # Verify headerscheck / cpluspluscheck succeed @@ -1015,12 +1068,19 @@ jobs: # - Use -fmax-errors, as particularly cpluspluscheck can be very verbose ### - name: headerscheck + cpluspluscheck - if: always() + if: ${{ !cancelled() }} run: | + echo "::group::configure" ./configure \ - ${LINUX_CONFIGURE_FEATURES} \ + ${{env.LINUX_CONFIGURE_FEATURES}} \ --cache gcc.cache \ --quiet \ CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" - make -s -j${BUILD_JOBS} clean - make -s -j${BUILD_JOBS} -k ${CHECKFLAGS} headerscheck cpluspluscheck EXTRAFLAGS='-fmax-errors=10' + echo "::endgroup::" + + make -s -j${{env.BUILD_JOBS}} clean + make -s -j${{env.BUILD_JOBS}} -k ${{env.CHECKFLAGS}} \ + headerscheck cpluspluscheck \ + EXTRAFLAGS='-fmax-errors=10' + + - *ccache_save_step -- 2.54.0.380.gc69baaf57b --rv3g7aw7ud36z5b5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6a-0003-disable-cirrus.patch" ^ permalink raw reply [nested|flat] 249+ messages in thread
end of thread, other threads:[~2026-06-01 19:09 UTC | newest] Thread overview: 249+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2019-12-20 01:25 [PATCH v29 11/11] Add documentations about Incremental View Maintenance Yugo Nagata <nagata@sraoss.co.jp> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de> 2026-06-01 19:09 [PATCH v6a 2/5] gha: Andres' revisions Andres Freund <andres@anarazel.de>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox